Comparison of PCA with ICA from data distribution perspective

ICA algorithm has a similar procedure to PCA in terms of minimizing the objective function and deriving optimal weights. We used iterative version of PCA and ICA exactly to be able to notice that fact. PCA has a property of ranking the basis vectors. The one that has the highest eigenvalue will correspond to the principal component which has the highest variance and which explains the largest part of the data and etc. ICA does not have such property. In fact, the algorithm finds a basis up to a scaling factor which means that a basis vector might be randomly multiplied by -1. ICA implies some assumptions on the distribution of data: all sources, except at most 1, should have non-gaussian distribution and thus it can be viewed as a generalization of PCA to non-gaussian data. It turns out that PCA is a part of ICA. It is used to whiten the data matrix before doing ICA which means that, since PCA works with first two moments of distribution, PCA and ICA do not intersect at all. Although, as mentioned by Aires, Chédin, and Nadal (2000), PCA might provide a good starting point for iterative algorithm of ICA. The main difference between them is that they are designed to achieve different goals: PCA is designed to maximize variance with certain ”bonuses” as orthogonality, linear independence and dimensionality reduction. On the other hand, ICA is used for separation of components (Jolliffe (2002)), extending this separation to a strict sense of independence than just uncorrelatedness, and it might fail to return reasonable outcome if there is no mixture (linear on non-linear) of independent sources in the data. In contrast to factor analysis, ICA does not try to explain correlations between factors. Rather it assumes that ecologically valid factors will be independent. Both techniques are not robust to noise which is the main weakness when they are applied to financial data. Although several variations exist (see for example Ikeda and Toyama (2000), Voss, Rademacher, and Belkin (2013), Bingham and Hyvärinen (2000), Candès, Li, Ma, and Wright (2011)) and PCA and ICA might actually be used for noise separation. In particular (based on their properties) ICA is best in non-Gaussian noise separation and PCA in Gaussian respectively.